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Add a sword and a cloak to the squirrel. 
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www.wix.com/blog/2016/10/10-photoshop-tips-and- 
tricks-for-beginners/ 
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www.wix.com/blog/ 2016 / 10 / 10 -photoshop-tips-and- 
tricks-for-beginners/ 
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www.engadget.com 
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Could we build models that use human 
language to automatically edit images? 
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Image Editing Pipeline 
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store.line.me 
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Image Editing Pipeline (Final Goal) 
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www.123rf.com 
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Image Editing Pipeline (Final Goal) 
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Add a sword and a 
cloak to the squirrel. 
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ImgEdit (Image Editing Request) Corpus M 
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Add a sword and a 
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ImgEdit (Image Editing Request) 


Corpus B 
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Source Image Target Image 



Collected from Internet 


Editing Request 


Add a sword and a 
cloak to the squirrel. 


Annotated by Human 
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Collect Images 
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Request/) 
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https://www.reddit.eom/r/PhotoshopRequest /13 



Collect Images 

^ Po5*' " " ■_E£:;le_lX 3 iiours r'n 

* [RANDOM] Please let me try this again mods. Can someone 
^ please add a bunch of SAILORS riding on the back of this 
seagull. It's a running joke for a group. Thank you. 


Reddit 

(r/Photoshop 

Request/) 



• 5 Comments A Share Q Save 0 Hide p Report 


^ Francophile_45 Sccr- .idden • 4 hours 
^ M v attemp t 

^ Blood_Eagle_lX Mddc" Ahoursigc 

^ Lol.. thanks! That's pretty good. 

imthatguyhere tr~gi icor-: " -dden 4 hours ::: 

^ Here'S a thing; 

https^/i.im a ur.com/iGvmOa d.onQ 

^ Blood_Eagle_lX ' ■ ■ >»ddf’" 4 hour. 

^ Thank you! Very., singing in the rain quality for some reason.... it reminds me of a 
movie from the 40s 
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Collect Images 

^ Poster uy 3 i»ours. “o 

* [RANDOM] Please let me try this again mods. Can someone 
^ please add a bunch of SAILORS riding on the back of this 
seagull. It's a running joke for a group. Thank you. 
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(r/Photoshop 

Request/) 



< ^ 3^01 e Mddc" 4 fwurs age 

^ Lol.. thanks! That's pretty good. 


imthatguyhere icor-: ^-dden 4rtours ::: 

^ Here'S a thing; 

https^/i.im a ur.com/iGvmOa d.onq 

^ Blood_Eagle_lX >»dd<’" 4 hour. ...c 

^ Thank you! Very., singing in the rain quality for some reason.... it reminds me of a 
movie from the 40s 
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Collect Images 


^ Poster uy 3 i»ours. “o 

* [RANDOM] Please let me try this again mods. Can someone 
^ please add a bunch of SAILORS riding on the back of this 
seagull. It's a running joke for a group. Thank you. 
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(r/Photoshop 

Request/) 
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Image 


P 5 Comments A Share Q Save 0 Hide p Report 

^ Francophile_45 Sccr- .idden • 4 hours age 
^ M v attemp t 

^ Blood_Eagle_lX ='-oie Mddc" 4 hour;:. 

^ Lol.. thanks! That's pretty good. 


imthatguyhere icor-: " dden 4 hours : 

^ Here'S a thing; 

https^/i.im a ur.com/iGvmOa d.onQ 


uddr" 4 hour. 


Target 

Image 


^ Blood_Eagle_lX 

^ Thank you! Very., singing in the rain quality for some reason.... it remmBs me of a 
movie from the 40s 
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zhopped.com 
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Zhopped + Reddit = 12K image pairs 

(r/Photoshop 

Request/) 


zhopped.com 
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Collect Editing Requests 


I 

* [RANDOM] Please let me try this again mods. Can someone 

* please add a bunch of SAILORS riding on the back of this 
seagull. It's a running joke for a group. Thank you. 


Redd it — 

(r/PhotoshopRequest/) 



• 5 Comments A Share Q Save 0 Hide p Report 


^ Francophile_45 Sccr- .idden • 4hours::s= 

^ M v attemp t 

^ Blood_Eagle_lX Mddc" Ahoursigc 

^ Lol.. thanks! That's pretty good. 

imthatguyhere 4 hours :: 

^ Here'S a thing; 

https^/i.im a ur.com/iGvmOa d.onQ 

^ Blood_Eagle_lX .r* ■ «»ddf’" 4hour. ... . 

^ Thank you! Very., singing in the rain quality for some reason.... it reminds me of a 
movie from the 40s 
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[RANDOM] Please let 


me try this again mods. 


Can someone please add 


a buneh of SAILORS 


riding on the baek of this 


seagull. It’s a running 


joke for a group. Thank 


you. 
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Collect Editing Requests 


♦ | - — - ^ - 

* [RANDOM] Please let me try this again mods. Can someone 

^ please add a bunch of SAILORS riding on the back of this 
seagull. It's a running joke for a group. Thank you. 


Redd it — 

(r/PhotoshopRequest/) 



• 5 Comments A Share Q Save 0 Hide p Report 


^ Francophile_45 Sccr- .idden • 4hours::s= 

^ M v attemp t 

^ Blood_Eagle_lX Mddc" 4 hours ajc 

^ Lol.. thanks! That's pretty good. 

imthatguyhere tr~gi icor-: " -dden 4 hours ::: 

^ Here'S a thing: 

https^/i.im a ur.com/iGvmOa d.onQ 

^ Blood_Eagle_lX ■ ■ >»ddf’" 4 hour. ...u 

^ Thank you! Very., singing in the rain quality for some reason.... it reminds me of a 
movie from the 40s 
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[RANDOM] Please let 


me try this again mods. 


Can someone please add 


a buneh of SAILORS 


riding on the baek of this 


seagull. It’s a running 


joke for a group. Thank 


you. 


Specifications are too 
noisy!! 
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seagull. 
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Collect Editing Requests 


Source 

Image 


Target 
Image 

12K image pairs 
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9K annotated 
image pairs 
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Correct 

Incorrect 


Request 


Add sailors on the 
seagull. 
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4K image pairs 

Correct 
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Incorrect 
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Task: Editing Request Execution 
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Add a sword and a / 
cloak to the squirrel. 
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Task: Editing Request Generation 
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Use Cases: 

• Explanation of complex image editing effects 
for laypersons 

• Visually-impaired users 

• Image edit or tutorial retrieval 
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Add a sword and a 
cloak to the squirrel. 
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Basic Captioning Model 


(a) Basic Model 
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Basic Captioning Model 


(a) Basic Model (b) Multi-Head Attention 
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Full Captioning Model 


(a) Basic Model (b) Multi-Head Attention 
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3 Evaluation Datasets: ImgEdit 
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Remove the man from the image. 
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3 Evaluation Datasets: Spot-the-Diff 
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The blue truek is no longer there. 


A car is approaching the parking lot from the right. 
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Jhamtani, Harsh, and Taylor Berg-Kirkpatrick. "Learning to describe 33 
differences between pairs of similar images.” EMNLP 2018. 
















3 Evaluation Datasets: NLVR2 
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Each image shows a row of dressed dogs posing 
with a cat that is also wearing some garment. 


h^ 

Adobe 


Suhr, Alane, et al. "A corpus for reasoning about natural 34 
language grounded in photographs." ACL 2019. 













3 Evaluation Datasets 

Ours (Image Editing Request) 


Spot-the-Diff 
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The blue truck is no longer there. 

A car is approaching the parking lot from the right. 


Add a sword and a cloak to the squirrel. 


NLVR2 Captioning 



NLVR2 Classification 


Convert 



Each image shows a row of 

dressed dogs posing with a cat —► True 

that is also wearing some garment. 

In at least one of the images, 

six dogs are posing for a picture, _► False 

while on a bench. 


Each image shows a row of dressed dogs posing 
with a cat that is also wearing some garment. 
















Evaluation Methods 


Phrase-based Metrics: 

BLEU, CIDEr, METEOR 

Human Evaluation: 

Pairwise Comparison 
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Phrase-based Metric: CIDEr 
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ImgEdit 


NLVR2 


Spot-the-Diff 



21 . 6 ^ 26.4 

(+4.8) 


43.4 ^ 46.4 
(+3.0) 


26.3 ^ 35.3 
(+9.0) 
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Human Evaluation: Winning Rate 
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11 % vs. 24% 24% vs. 37% 


22% vs. 37% 
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Generated Results 


Image Editing Request 


Positive 

Examples 




change the background to blue 


Negative 

Examples 



add a filter to the image 



UNC 

NLP 


h^ 

Adobe 


Spot-the-Diff 



the person in the white shirt is gone 



the black car in the middle row is gone 


NLVR2 



there is a bookshelf with a white 
shelf in one of the images . 



the left image shows a pair of 
shoes wearing a pair of shoes . 
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Code at: 

https://github.com/airsplavA/isualRelationships 
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